Add Triton + TensorRT-LLM inference example #86
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
milo157
reviewed
Nov 19, 2025
|
|
||
| input [ | ||
| { | ||
| name: "text_input" |
Author
There was a problem hiding this comment.
This is only for configuring the shapes and other params. System prompt would have to come directly in the model.py
| #!/usr/bin/env python3 | ||
| """ | ||
| Download HuggingFace model to persistent storage. | ||
| Only downloads if model doesn't already exist. |
Contributor
There was a problem hiding this comment.
Do you run this with cerebrium run or does it run on deploy?
Author
There was a problem hiding this comment.
I thought about this but I'm also not feeling great on reinstalling packages (once through the docker file and once through toml)
| """ | ||
| responses = [] | ||
|
|
||
| for request in requests: |
Contributor
There was a problem hiding this comment.
This doesn't look that efficient
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This example demonstrates how to deploy a high-performance LLM inference server using NVIDIA Triton Inference Server with TensorRT-LLM for optimized GPU inference.
Features:
The deployment uses an A10 GPU and exposes Triton's standard HTTP API on port 8000.